Multiresolutional descriptions of digital video in terms of "relevance"

This paper introduces algorithms for determining the “relevance” of each frame (and objects within the frame) to the scenes present in a given video sequence. Specifically, we introduce a rank number which determines the importance of each frame in contributing to the scenes in a video sequence. The rank number is computed based on the temporal duration of motion-tracked objects and pixels in video frames. Using this rank number we show that multiresolutional descriptions of the video sequence based on hierarchically selected temporal segments and key-frames can be constructed. In order to motivate the use of our approach in video summarization and content analysis we argue that pro-minent objects, those that appear for extended periods of time, are usually deemed more related to the general contents of digital video. It is discussed that objects and frames appearing briefly can also be quite important in helping determine the contents of a video sequence, necessitating a ranking mechanism. Coupled with a temporal segmentation algorithm that is also based on tracked pixels, our results involve multiresolutional key-framing and temporal segmentation techniques that can be used to analyze and browse video sequences.